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sponsorship, and (d) the round-figure magnitude ol the program 
(i.e., number of pr6ject<;^ participating school districts, sponsors,, 
and SO' on) . 

2. Describe the evaluation design against the backdrop of program 
development. 



3, Identify live issues that were reconciled one way^ but might have 
been reconciled in other ways, and note how these decisipris have 
influenceci the evaluation 6j F^ Through* 
In the discusfiion later, I hope there will be ah opportun^^^^ to sugge^^t 
.some icssohs thatFol low Through has provided for planning and carrying out 
ifjtture social exporinHnts in naturaT settings • 
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yTSaio^^^^^ Throuyih Program - ^ 

- " . _ _ " - _ " _ _ _ -I - 

_ Congress authorized Follow Tnx'ough in 1967 «Mcr ah amendnieiit to the 

/feconomic Opportunity Act to provide devc\ippmental and educational services ; 

|or j)oor childi'cn in primary grades who had experienced Head Start or equiyaa 

l^tii pi^escbool . A- large-scale service program, roughly similar in scope to 

>ltead Start, v.'as cnvisibhed originally and reflected in the enabling legisla^ 

$ion. Appropriations v;ere not sufficient, however, so the program was re^casit 

as an H&D program even though the language of the legislationi itso-Lf-was 

hever ch.angcd. This IIU) orientation has had complex effects on the way the 

Vprogram (and its evaluation) evolved and how various staUeholderr. view .and 

pudge the program. 

Since Follow T)n*ough was authorlsied under the EGA, the Office of Ecohomi:<i 
?Spport unity had rospon;.?ihll j ty for it just as they did for Head Star-I:. CEO 
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PA* Rfogram priorities. Also, every individual project applies 
ijome additional variations on sponsor's objectives that will be 
sought locally. The measurement" effort necessary to embrace the 
full sweep of all objectives, hcld.^by^ .all sponso^^ individual 
projects has "never yielded gracefully to solution. As a consequence, 
it is ine\dtabj£j;h^ obtained can-be critici^iiiiLji^ 



sonie justification by evei^ sponsor or project as not including, 
• "all the things 1 am trying to do," 

Gvervicv/ of Evai 

The basic design is tirat ^of a before --and-a Iter Follow 
Thi^ugh progi-am ^p define the experimental groups ajid nonpar ti<fi_T 

pants of similar characteristics ^comprise the controls, "Before' (in the^ 
beiorc-ahd-after d^^^^^^ refers to measures dbta^^^^^^ as possible. ig: 

the initiation of the experimental treatment. This, htus meant m^ 
pupil characteris tier as close to the beginning of a c^^^^^^^^ 

-year as could be mdhagod administratively. "Before" measures on other pngici- 
ipants, such as parents and teachers , have usually beeii .obtained at a later 
;point in the first year of school. 

The final Sf tor" point cantTot yet be describe Historically, During^ 
early planning in 1968-69V th^- intended "a£ter" was a point as= 

jiear the end of the child's third Grade as could .be managed, By a decision 
adopted in Spring 1970, children entering their first school year .of FoUgi 
Through in Fall 1969 formally became the first cohort eligible for evaluatron;. 
children who entered in Fall 1968 were officially acsighated as the 
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CM • ■ . 

Wn've drawn t-osollic)- by coinir.ou inlorervts in oaviy childhood education 
or evaUiotion or both. Probably oach o/ us carries r.n nxe hc-'d liKe to 
;'%\ <>riJid. or even swin^^. 

O I'd like to be a lens grind&r instead of an irxo grinder, at least in 

O these oj.onir:- remarks. Since my perspective on boih early childhood odncn-- 
lio-.i :'.u! the evaluation of earjy childhood programs has boon so totally 
ck.r.:jn-Jl^d by responsibilities in the Follow Through program smco July 3960, 
the Jews I'll try l.o grind is one- t)ivoui;h which we can look at the Follow 
Throii.^1: pvoi^rnm and its evaixiation. 1 want to locus on a scloci-.ed few 
issues that have intJuenced the evolution of both the FoUov; ThX-oufch ;»-0}.t. am 
and iis evaluation. My purpose is ro provide a perspective from v.-hieh con- 
struci^^ve asses^-Monl. of ihe Fo.lJow Through evaluation can be made. (■^Ve car. 
fcriud and swing our axes in the iater give-and-take.) 
My remarks cover the lolio'.vinii: 

1. Give a b)-ici overview of Foliow Through (on the as:.unnpl ion that 
noL an those present are familiar with il). This overview wiJ I 
include (n) initintjon of the program, (b) transition irom a 
v:ervic'- '.o R&)» oi icui rM. i on , (c) the concept of pro;^ram 



.sponsorship, and (cl) llio roumi-.l'i ijuru inagniliuio ol the proj^i'am 
(i.e., number of pr(.joct<:, pa rtici patinj!: school dislrJcis, sponsors, 
and so on) . 

2. Describe the evalualion dei?l^rt nj^riinst the backdrop of program 
development . 

3. IdentiXy live issi:os that wore reconciled one way, but h-ight have 
been reconciled in other ways, and note how these decisions have 
inl'lueiiccci the evaluation of Follow Throu^^b . 

In the discuriiiion later, 1 hope there will be an opportunity to suj^t»est 
some lcst:ons that Follov/ Throuj^h has provided for planning and carry.inp: out 
future social expcrlmcp.tis in natural settings. 

Vol 1 o 'rim) u p^ P rogra m 

Congx-oss authorized Follow Tnrou|di in 1967 under an aniendmenl" to the 
Ecoiioiuic Opportunity Act to provide dovelopiriontal and educational services 
for poor chr.Idren in prima)-y jjjrades who had experienced Head Sta.vt or equiva- 
Ici^i preschool. A larj^e-scaJe service proj^ram, rou(rhly similar in scope to 
Head Start, was envisrloned or1<cinally and reflected in the enabling; legisla- 
tion. Appropriations were not sufficient, however, so the pro<{ram uas rc-cast 
as an RiM) proj^ram even thouj^ih the lan[^uaf,-G of the legislation ilself was 
never c1>arii.Acd. This RLD orientation has had complex effects on the way tlie 
proi.;'j-am (imkI its evaluation) evolved and how various stakebolderj-. view .and 
jufh^e the pj*oj.vram. 

Sin<:e Follow Through was authorised under the EGA, the Office of Kconomic 
Opportunity had respoaslhil 5 ty ior it just as tl)oy did for Head Start. Oi::o 



trans fci'red funds to the Ofl'ico oX Education foi" program administration, 
but GEO retained a vested interest in the progi'am*s Ue^i^jn ar.d management. 

Follow Through bc?gan in the* 1967-68 school year when 52 projects were 
initiated iii 40 pui-pcsively selected school districts. All these first 
projects wore of the type that have come to be called "self -sponsored ; " 
that is, each local district conceived and began to ampl.emcnt its own 
concept of a "best" or most appropriate program for impoverished primary 
grade children through grade three. This notion of intentionally diverse 
approaches was labeled "planned variation," 

purihg 1967-68, the Follow Thi^ough progri^m office in OE refined the 
concept of planned variation: to more nearly approximate a systematic experi- 
ment. Various recognized proponents of early education approaches were 
invivted to serve as "model sponsors" to install and support their approaches 
in one or more project locations. This mode oX sponiiorshii began with the 
1968r69 school 3'eur, Local preforonces for models or approaches were 
honored -- districts chose their preferred approach (within loose limits) 
following a sort of "courtship" with intere.sted sponsors. Note well that 
principles of local autonomy and mutual preference determined the sponsor- 
to-rdistric:t affiliation rather than some criteria of experimental design. 
At the start of the 1968-69 school year, there were lOG projects and 15 
sponsors, cvlunting self-sponsored and parent implemented projects as "sponsor 
categories. About half of the projects initiated in 1967-68 affiliated with 
a sponsor and the remainder continued .as self-sponsored projects. 

Some projects were added in 1969-70, bringing the total number 16 160- 
In- add:i tion, six now sponsors joined the effort. More sponsoi-s and a few 



m(iro projects were added jn 1970-71 > bringing the total projects to 177 and 

the number of sponsors to 22* A few more projectr* and two more spom;ors were 

added in 1971-72 to bring the project total to its current level of 180 and 

sponsor total to 24 (ignoring, for simplification, a sponsor who opted oiit 

in 1969 --70 and a few projects that also elected to terminate in 1969-70). 

A bo u t ^ 9 ' ^. 000 i ihiX ilx iLn tire involved in 3*'ollow Througli_this y ,eax^>^ 

The following are points to remember about Follow Through as v/e appraise 

< 

Its evaluation: 

!• the selection of districts, schools, teachers, and childx-en to 
be involved in Follow Through was judgmental and guided by rules 
of individual district or child eligibility (e*g. , districts in 
poor communities i children f x'oni families ^that were poor by OEO 
poverty guidelines). None of the choices (with the possible 
exception of some participant selection within a project) were 
raiidom. 

2. Control groups were not established simultaneously with the expei^d- 
mcntal groups (i.e., PoXlow Through participants) by random 
selection from pools of children eligible for participation. All 
control groupr5 were established after-the-fact. 

3. The mo u by which sponsors affiliated with school districts was 
purposive in the extreme and led to substantial imbalance in ^--^ 
characteristics across projects within sponsor categories and between 
sponsor groupings . 

4. I've not mentioned it specifically yet, but you can appreciate 
that each sponsor entertains a somewhat different set of objectives 



or program priorities. Also, every individual project applies 
yortio additionnl variations on sponsor's objectives that will be 
souglit locally. The measurc-mont effort necessary to embrace the 
full sweep of all objectives held by all sponsors and all individual 
projects has never yielded gracefully to solution. As a consequence, 
it is i ne vi tablc^tha t jLhc mea su res obtained can-bo-criticiJ5xiji.J;^^ 



some justification by every sponsor or project as not including 
• "all the things I am trying to do." 

Gvorvicv/ of Evaluatio n Design 

The basic design is that of a before-and-af ter experiment. Follow 
Through program participants define the experimental groups and nonpnrtici- 
pants of similar characteristics comprise the controls, "Before" (in the 
before-and-after design) refers to measures obtained as near as possible $o 
the initiation of the experimental treatment. This has meant mea.'nircs of 
pupil characteristic- as close to the beginning of a child^s entering school 
your as could be managed administratively. "Before" measures on other partici- 
pants, such as parents and teachers, have usually been obtained at a later 
point in the fir at year of school. 

The final * .fter" point cannot yet be described historically. jDiiring. 
early plannin!,' j.n 1968-69, the intended meaning of "fiftor" was a point as 
near the end of the child's Third Grade as could be managed. By a decision 
adopted in Spring 1970, children entej-lng their first school year of Follow 
through in Fall 1969 formally became the first cohort eligible for evaluation; 
children who entered in Fail 196S were officially designated as the 



"implcmcntat J.ou cohort'^ and therenXtcr cxciudcd Ixom the; evaluation. By 
this convention, the fii'st genuine "graduates'* of Follow Thx'ouj^h arc children 
who began Follow Throu^xh as First Graders in Fa 13 1969 and completed Third 
Grade in Spring 1972. Children who began Follov,- Through as Kindorgartners 
ir. Fall 1969 will not complete Third Grade until Spring 1973* Current 
expectations are that four successive cohorts will be iollowed longitudinally. 
The lourth cohort entering Kindergarten in Fall 1972 will complete Grade 
Three in. Spring 1976. 

Major classi.Cicatory variables for analy^^es include (1) cohort (one 
fhx'oiigh four) , (2) grade r5treani (enter kinderg^irten o.r first graded, a»d 
(3) treatment (Follow Through vs, non-Foilow Through within sponsor or approach 
categories and approach vs. approach according to differences between Follow 
Through and hon-Fdllow Through) , A variety of other independent variables 
iax'e employed in the analyses , biit it is not appropriate to discuss the 
=detnil of analyses or findings to date in this symposium. 

Two additional design considerations should be noted, for they are , 
critical to analyses, iuterpi*etationr> of findings arid suggest lessons for 
future planning. 

One issue Is the frequency and coverage of measures intermediate between 
the "before" o;. entrance measures and the "after** or exit measures. Inter- 
mediate level measures have been obtained for a subset of projects oh which 
entrance measures were taken ho that repeated measui*es would be available on 
chj]dren in some projects. The number of projects in which intervening time 
measures are available is smnJI relative to the number of projects with 
busG line measures. The decision to limit the number of projects to be included 



ill data collti^tion at intcnaediatc points for any one cohort of pupils was 
partly o matter of economy (and deference to the tolerance of individual 
projects for irequent across-the-board measurement). More si^^nif icantly , 
this limitation provides an effective restraint upon the temptation to drav; 
conclusions about the efficacy of an approach before two conditions can be 
satisfied: (1) completion of a full term of three or four years for a 
cohort group and (2) accumulation of a reasonable number of examples (i.e., 
projects) ix-prcscnting an approach. 

The second issue is that of control groups. I noted that these were 
es:tublished after-the-fact for each project and not randomly assigned when 
cxperj mental groups were created. This condition constrains statistical 
inference. In addition, these non-Follpw Through control.'? often are quite 
dissimilar from the Follow Through samples on many key demographic and socior 
economic indices. I will dismiss, as mess> but manageable, the administrative 
and diplorr.ntic problems of finding and inducing schools to collaborate a^:- 
-controls. , 

I hope you can see, from these comments, that Follow Through should 
not be viewed af; nn experiment in the classical sense. From the evaluator*s 
viewpoint, it i.': a quasi- or even a pseudp-expc^rimcnt in which thei^e is 
literally no m: .ipulative control of treatment.^. It's social experimcntatidn^ 
oh a fairly ambitious scale with all the blooming, buziiing confusion that 
characterizes efforts to be systematic in a natural setting-. 
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Somo Key issuer Wi th Alter nate Hos olutloiifi 



The Service Orientation vs. the Research Orientation 



A service orientation toward Follow Throuf^Ii would dictate aj5.suring that 
only the most needy were participants in the prep:ram. In conti'ast, an 
experimental orientation would either a<;sure equal needuiess for participants 
and non-par tic: pants or would permit variation on the inipoverishment scale 



so that a "treatment by poverty** interaction might be identified. A service 
orientation also would argue for a standard presentation of services to all 
who qualify, whereas an experimental orientation would encourage a greater 
variety of services. Fina^Jy, evaluating a program under a service orienta- 
tion would require some pro-defined standards against which program success 
could he measured. In contrast, an exper iinental orientation would moi*e 
lihely ask Nvhother iiiter-treatmcnt differences existed, and if so, where, 
to what degree, and so on. ^ 

Officially, Follow Through is an R&D program. Many people appraise it, 
however, in a service program context. In addition, many of the operational 
decisions , especially at local project levels, have been made as though the 
program were clear 1\' and certainlj' a compensatory service program. Most of 
these latter decisions have been ones that juggled and reassigned participants 
according to Judgments about the needs of individual children. 

Polic y vs . T heo ry O r ientation 

The grossest possible question that the policy maker might askis **boe;s 
it work?** The theoretician is more likely to bo concerned with the conditions 
under v/hich particular effects arc observed; i.e., the how, why, and wholly 
(iueytlons. An analoglcaJ distinction between the two orientations cati be 




made by rcferi^ing to iho input-black box-output model Xaniliar to operations 
research, A policy orientation would sugiccst contrasting input levelj^J with 
output levolfj without necessarily addressing qucstionr. oX what happens inside 
the black box. The theory orientation, while not disinterested in input-output 
dif j.'eronces , Is particu3arly concerned with the processes inside the black 
box. A policy orientation also is more likely to ask cost effectiveness 

-find cos t bene 1 i toque's tiolii of the "cTaTa , wlrei^eivs- -44ie -tlieoxx^li^yit^tJjm-Jria^^ 

in many cases, ignore the cost variable. 

The importance of this policy vs. theory or research orientation is ho 
more sharply Xelt than wlicn one speaks to a mixed audience of policy makers 
or administrators on the one hand and, on the other, researchers who address 
questions cf theories of instruction or human development. The "grand pian'* 
fpr the evaluation was one that anticipated patience by policy makers and 
legislators wl»ile some how and why questions were pursued. We have expeiU.- 
ehced increasing pre'ssurcj, however, to draw policy-type conclusions on the 
bjisi{5 of incomplete evidence. They ask serious and legitimate questions 
that deserve sober answers — such £\s, "Should Follow Through Approach X be 
cpntimicd?** — but the evidence is partial, often inconsistent, and sometrmejs 
appears trivial. Policy makers shew occasional impatjence with answers like 
"just wajt four more years and spend another umpteen dollars and we ought 
^to be able to make a better estimate," 

For mati ve vs . gumm a tivo Orientatio n 

If one gives prominence to formative asser^sment, he is by that choice 
encouraging the program to cliangc as it grows in response to frequent and 
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faix'ly rapid feedback. Sunmu'itlvc assessment, on the other hand, is more 
congenial to a stable tre- ^mcnl observed over a sulficicnt period to pei*mit 
conclusions to be drawn about the whole program or the I'olative strength of 
fixed alternatives. These comments are not meant to imply that one view of 
assessment is "better" than a-.other but sinply to note that formative 
assessment is most appi'opriate for those conditions that obtain when designing 
a system is the primary oJyju&t±v(ri-^'n& summa ^±vkl iHss^se 'Ssment is most appi'opri- 
ate to conditions where the ob.ipctive is to tef^t the worth of a describablc 
system. 

In the first year or tv;o of the evaluation, It was felt that both 
orientations could be maintajned simultaneously that a single agency could 
provide rapid turnaround of data to guide pi'ogram design decisions and also 
manage the detachment and objectivity required for summativo assessment. 
The load was more than we could manage, not due so much to data processing 
iahd turnaround time as with the differences in the kinds of data appropriate 
for the differing classes of questions. There's a fundamental incompatibility 
between emergent treatments, influenced by formative assessments, and experi- 
mental treatments that maintain consistency long enough to warrant inferences 
being drawn about their effects. 

General vs , Specific Crit eria of Success 

This dichotomy might !iave been referred to better as abstract vs, 
concrete, or less measurable vs, more measurable outcomes, I don't know how 
to state a ciear pi-cferencc with i-egard to broad vs* narrow criteria or 
aggregated and gross outcome measures vs. disaggregated and fine-grain 



inoasiircs. I)i a sciiHo, it is a no-win gafne, Somo of the approaches in 
Fql-lov/ Through Jjavc never managed to state their objectives in more than 
very Inroad terms, thus creating difficult, if not insoluble, problems of 
measurement. On the other hajid, those that have stated spcicific objectives 
in explicit terms (which icnd themselves bottex' to measurement) may be 
overemphasi;iing educationally trivial things. An objective of "increasing 



the lile chances of poor children is both noble and general , abstract , 
nonspecific, and uhmoasurable. An objective of "teaching children to count 
dots," while defensible as an instrumental objective building toward 
cbinpetence in concepts of number (and therefore maybe contributing to future 
life chances) is a rather narrow criterion for judging a progi'am. The golden 
mean is somcv/here in between. 

Frequent Re porting vs , Deferred Reporting . 

This issue is related to an earlier one of fox'mative vs, summative 
asscr^sment and olso to tlie policy vs, cheoretieal orientation. Certainly 
the confidence that one may express about observed findings is a function 
of the reliability of the measures and reliability is greater with repeated 
measures over aw extended period. In Follow Through, the heart of the issue 
on frcqtjent rep rts vs, deferred reports is whether all of the Follow Through 
approaches are going to be given a full period during which each of several 
cohorts can display their effectiveness. If comparajtive findings are 
reported frequently, tlien the risk that someone will draw premature con-^ 
elusions is increased. On the other hand, if Teports are deferred too long 
or qualified excessively when issued^ their utility for the policy maker may 
be/ lost entirely, 
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Pcrhaps Ihu ba]anco can bo struck by acknowledj^inj^ the legitimacy of 
two dis tinguishP.bJo claspos of reports. One would emphasize timoJinos.s and 
bcs t-estin.r;te trends to reduce the chances of f:ross errors of either omission 
or commission by poJicy makers. The other kind of r-Moa " ^iuld contain more 
definitive analyses that sought to explain pheuomeu.. , .rhaps only partly 
perceived in reports whej'e timeliness overrode precision in importance. 
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